Make the OpenSearch max_result_window raise take effect after the first time - #1846

Problem

The background index sync raises index.max_result_window to 1.2× the indexed document count once an index exceeds 10,000 documents, so from + size paging can reach everything indexed. But OpenSearch.putSettings() sent that PUT with ?preserve_existing=true, which OpenSearch defines as "leave any setting that already exists unchanged". The window was therefore set exactly once in an index's life, the first sync pass after it crossed 10,000 documents, and never raised again.

SearchApi answers HTTP 400 for from + size beyond the window, so API and token clients, and the UI paginator (which reads X-Wildbook-Max-Result-Window), were capped at a stale number.

Measured live on 2026-11-04 before the hand fix:

The encounter windows on both installs decode to ≈12,300 documents: an index filled by the background sync in slices of 2,500 first exceeds 10,000 at ≈12,500, and that first pass was the only one that ever stuck.

Fix

Single file plus its test.

• putSettings: drop preserve_existing. The only other caller is createIndex(), which sets the analysis normalizer on a just-created, closed index; Wildbook's normalizer definition is authoritative, so a template-supplied one being overwritten is by design (noted in a comment).
• targetMaxResultWindow(count): the sizing rule as a pure function (1.2× the count, −1 at or below the default, clamped to int).
• ensureMaxResultWindow(index, count): grow-only. Reads the stored value fresh and strictly, never through the 60 s cache or the tolerant parseMaxResultWindow default, and skips the update on any read failure (transport, response shape, non-integer value). A transient OpenSearch error can therefore never lower a window, including one an operator raised by hand. After a successful PUT it drops the cache entry so the API gate sees the new window at once.
• getAllVersions calls ensureMaxResultWindow instead of the inline PUT. A failed PUT still propagates and aborts the sync pass, exactly as before.

Not changed: the scroll, the sync cadence and slice, the 1.2 factor, the 10,000 threshold, the API gate.

Tests

OpenSearchPaginationTest, 12 new cases, all written first and watched fail:

• sizing rule: 12,187 → 14,624 (the stale field value), 544,735 → 653,682, 10,001 → 12,001; ≤10,000, 0 and negative → −1; clamps at Integer.MAX_VALUE for int and long overflow
• through the real putSettings (Mockito spy on getRestResponse): raises a smaller stored window with endpoint encounter/_settings and no query parameters; never lowers a larger stored window; skips when the settings read throws, when the response is malformed, and when the stored value is not an integer; treats an absent key as the default; sends no traffic at all for an index at or below 10,000
• the decision reads the live setting, not the cache: a primed stale value plus a larger live value produces no PUT (mutation-checked: pointing the decision at the cached reader fails this test)
• the cache is refreshed immediately after a PUT

Full suite run locally in the worktree: Tests run: 1832, Failures: 0, Errors: 0, Skipped: 7, build success.

Operations note

This PR stops the regression on the next index rebuild. Indexes that are already stuck need a one-time hand fix, which has been applied on flukebook.org and mantamatcher.org:

curl -s -XPUT 'http://localhost:9200/<index>/_settings' -H 'Content-Type: application/json' \
  -d '{"index.max_result_window": <1.2 × _count, rounded up>}'

Size from GET <index>/_count, not _cat/indices, whose docs.count includes nested documents (individual names, annotation embeddings) and overstates by 3–7×. Other installs whose indexes grew past 10,000 documents after creation have the same cap until the next sync pass with this fix deployed.

Review

Plan and code reviewed by Codex (OpenAI). Its plan review rejected a first grow-only design that compared against the cached, defaulting reader, because a failed read would have lowered a hand-raised window; the strict uncached read and the three read-failure tests come from that round. Its code review converged with no Major findings; the three Minors (non-atomic write assumption, normalizer-template note, stale-cache regression test) are addressed in the second commit.

🤖 Generated with Claude Code